m m 



CMLOOOIOH 



TEXT INPUT METHOD 
FOR PERSONAL DIGITAL ASSISTANTS AND THE LIKE 

DESCRIPTION 
RELATED APPLICATION 

The presem'^gplication is related to U.S. Patent Application No. 09/ 

(Attorney Docket No. LXl)©Q71) entitled "Handwriting User Interface for Personal Digital 
Assistants and the Like" to Senie^aL assigned to the assignee of the present invention 
and filed (either coincident or prior), 

BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention is related to personal digital assistants (PDAs) and more 
particularly to a user input interface for a PDA or the like. 

Background Description 

Portable computing devices, such as what is normally referred to as a personal 
digital assistant (PDA), are increasing in popularity. A typical PDA is a limited function 
microcomputer provided with a pressure sensitive liquid crystal diode (LCD) display (a 
touch pad or a touch screen) for input and output (I/O). PDAs are being adapted for 
wireless Intemet communication, e.g., using a modem for e-mail and web browsing. 
Further, for text input PDAs are known that have a specialized stroke based alphabet 
interface, e.g., Graffiti®, a selectable on-screen QWERTY keypad, or an expansion pack 
keyboard. 
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As these portable devices become smaller and more specialized, text input has 
become more difficult and less practical. Typical prior art handwriting recognition 
software may require users to learn special characters or effect a handwriting style in 
order to enter text. Text input using the Graffiti® unistroke (i.e., written with a single pen 
trace) alphabet can be un-natural because it requires users to adhere to strict rules that 
restrict character shapes; text input using an on-screen QWERTY keypad is somewhat 
clumsy because only small reductions in size can be made to keyboards before they 
become awkward to use. An expansion keyboard is impractical for on-the-go input. With 
either, the tapping on individual characters or the typing is less desirable than being able 
to handwrite notes or messages. Meanwhile, the demand for PDA information exchange, 
e-mail and internet access requires entry and retrieval of increasing amounts of data with 
the handheld device. 

Natural handwriting recognition (HWR) programs have been developed to add to 
function and usefulness to PDAs and are crucial to the growth of mobile computing in the 
communications field. To use handwriting recognition software, such as Transcriber 
(formerly known as CalliGrapher) fi"om Microsoft Corporation, the user writes a message 
anywhere on the PDA screen, i.e., on top of any displayed application and system 
elements. Altematively, the user can write in a designated input area using handwriting 
recognition software such as QuickPrintPro™ firom Motorola, Inc., Lexicus division. 

. In order to handle unconstrained handwritten input (i.e., written in cursive style, 
print style, or using a combination of both) adequately, typically HWR programs such as 
those mentioned above, employ recognition algorithms that rely on dictionaries to 
constrain the search decoding space — i.e., the space of possible letter sequences that can 
be matched to a given input ink. Users, however, often need to write otherwise non- 
sensical words (e.g. an e-mail address) that are outside of a typical system dictionary. For 
these situations the dictionary is useless and imder some circumstances using a dictionary 
may actually impede correct recognition. However, HWR systems that are not, at least in 
part, dictionary based cannot support cursive writing where character boundary 
information is not available. Systems that are capable of generating both in and out of 



CMLOOOIOH 



3 



dictionary recognition hypotheses simultaneously, typically exhibit lower in-dictionary 
accuracy. 

Thus, there is a need for handwriting input user interface that includes the 
advantages of providing a dictionary for normal written word recognition, while at the 
same time allows users to enter any out-of- vocabulary character string. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, aspects and advantages will be better understood 
from the following detailed preferred embodiment description with reference to the 
drawings, in which: 

Figure 1 shows a preferred embodiment handheld device with a user interface that 
includes separate handwriting input areas for words and for characters according to 
preferred embodiment of the present invention; 

Figures 2A-B show handwritten text entry and recognition, wherein handwritten 
word entry begins in the word input area of the user interface; 

Figures 3 A-B show individual character entry and recognition in a character entry 
area of the user interface; and 

Figure 4 is an example of a flow diagram for implementing the handwriting user 
interface of the preferred embodiment of the present invention. 
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DESCRIPTION OF PREFERRED EMBODIMENTS 
OF THE INVENTION 

The present invention is a method of interfacing with and, a handwriting user 
interface (HUI) for, small (shirt pocket sized) portable devices with a touch-enabled 
input/output (I/O) screen, such as are commonly known as personal digital assistants 
(PDAs). The portable devices may be capable of wireless message transmission (such as 
for web browsing and/or e-mail). The user interface of the present invention is typically 
in software and loaded into storage in PDA. A state of the art handwriting recognition 
engine may be included, also in software. The present invention is particularly 
advantageous to entry of text, including character strings that are not necessarily words in 
a standard sense, e.g. acronyms, nmenonics or uniform resource locators (URLs). An 
entire character string, e.g. a URL, may be quickly handwritten, converted to ASCII text 
and, possibly, stored in the user dictionary for future reference, for example. Thus, the 
present invention enhances the usability, flexibility and power of the handheld device in 
which it is installed. 

U.S. Patent Application No. 09/ (Attorney Docket No. LX00071 

entitled "Handwriting User Interface for Personal Digital Assistants and the Like" to Seni 
et al., assigned to the assignee of the present invention, filed (either coincident or prior) 
and incorporated herein by reference, discloses a handwriting recognition user interface 
(HUT) which may be combined with the present invention for receiving text based 
handwritten entry. Handwritten entries are made at a designated input area on the touch 
screen, e.g., dimensions 0.30*H by W, where H and W are the height and width at the 
bottom of the device screen. Handwritten words are entered into the designated input area 
one at a time using a stylus. Recognition results are displayed in the normal display area 
of the screen above the designated input area. 

Referring to Figure 1, a preferred handheld device 100 with a graphical user 
interface for entering handwritten text 102 according to the present invention is shown. 
Most preferably, the present invention is implemented in the user interface of Seni et al. 
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mentioned above, further designating respective word and character input areas 104, 106 
within the designated input area 108 described therein. As shown, this input area 108 
preferably resides on the lower third of the screen 102 with the area 109 thereabove 
reserved for display of screens associated with the device's various applications. Icons 
110, 112, 114, 116 and 118 are disposed, for example, at the right side of the handwriting 
user interface 102 adjacent the input area 108. The icons 1 10, 112, 1 14, 1 16 and 118 
provide access to functions such as inserting a space, backspacing, deleting, capitalizing 
recognition result, and undoing insertion of a last word recognition result. A scroll bar 
120 is disposed, for example, at the right side of the interface display 102 adjacent the 
screen application area 109. 

An entry that begins in the word input area 104 is treated as a handwritten word 
and recognition proceeds as described in Seni et al. However, when a handwritten entry 
begins in the character input area 106, that entry is treated as a single character and may 
be one character in a character string. Handwritten character entries are each matched 
against potential characters, e.g. characters in the particular PDA character set. 

The device 100 may include a communications function and, to that end in this 
embodiment, an antenna 122 is shown at the top of the device 100. Individual function 
switches buttons and other controls are disposed about the device, as is deemed 
appropriate for the particular device. Preferably, the device 100 runs under a state of the 
art operating system for such handheld devices, e.g. Windows® CE from Microsoft 
Corporation, Epoc® from Symbian or the Palm OS® from Palm, Inc. 

The preferred embodiment HUI of the present invention employs a handwriting 
recognition engine capable of recognizing handwritten words, written using any 
combination of writing styles (i.e., cursive, print, and mixed). Preferably, the recognition 
engine is the QuickPrintPro™ engine from Motorola, Inc., Lexicus Division. The 
recognition engine typically includes a main word dictionary and one or more user 
dictionaries to which the user may add words to supplement the main dictionary. User 
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dictionaries may include, for example, a word list, an e-mail address book, a 
bookmark/shortcut list, etc. 

Figures 2A-B show handwritten text entry, wherein text entry begins in the word 
input area 104 in Figure 2 A causing the entry to be recognized as a handwritten word and 
treated accordingly. To make a wider writing area available for word entry, upon first 
pen-down within the word input area, the character input area is removed and 
subsequently restored after completion of the word recognition operation. Thus, in Figure 
2B after entry is complete, word candidates are presented in a pop-up list 130. Location 
of the pop-up list is not critical and so, to conserve space on the screen 102, the pop-up 
word list 130 can be displayed in the input area 108, and more specifically in the text or 
word input area 104 thereof 

The recognition engine compares a handwritten input word against all words 
contained in the main dictionary and the user dictionary. A probability score is generated 
by the recognition engine for each dictionary word which is indicative of the likelihood 
that the handwritten word matches that particular dictionary word. Based on each words' 
probability score, a list of likely matches is collected. From the recognition results, the 
handwriting recognition engine generates a confidence score for the one word (the 
primary word) with the highest probability score. If that confidence score exceeds a 
preselected confidence threshold, it is an indication that the recognition engine has 
correctly recognized the written word and that the primary word choice is in fact correct. 

So, if the confidence level is above the preselected threshold, the HUI 
automatically loads a primary word choice into the device's input buffer for delivery to 
the active application. Thus, in the example of Figures 2A-B, that word may be 
automatically inserted into a displayed stream of recognized text such as in the upper area 
1 3 1 of the display screen 1 02 in the application area 1 09 thereof. Otherwise, when the 
confidence level of the primary word choice is below the confidence threshold, an 
indication is provided that the recognition engine cannot find a likely candidate, e.g., 
displaying "???" or something similar into the device's input buffer. If the correct word is 
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not in the pop-up list the user can either rewrite the word or "back-off to character-by- 
character input in the character writing area. 

Handwritten text input entry may be provided in unrestricted mixed style that 
includes cursive (i.e., contiguous characters in each word touching or connected), pure 
print (i.e., characters in every word disconnected and do not touch), pseudo-print (at most 
pairs of characters in words touch) or any combination thereof. Thus, for mixed entry a 
user is not restricted to cursive, print or pseudo-print entry. The user may designate that 
text entry is to be in one mode only, i.e., cursive, pure print or pseudo print. Thus, the 
number and complexity of created character altematives possible are reduced for the 
handwriting recognition engine, increasing both recognition accuracy and speed. 

Figures 3 A-B show individual character entry wherein characters are entered in 
the character input area 106. Individual characters may be entered as part of a non-textual 
string (e.g., an e-mail address) or to correct a misspelled previously entered word 
identified by tapping on the word to select it, for example. Preferably the character 
writing area is defined as "modal" such that the possibilities of the user's character input 
are selectively limited in order to increase recognition accuracy. Selectable modes may 
include "digits," "symbols," "upper-case letters" and "lower-case letters" modes. 

Entry begins in the character input area 106 in Figure 3 A, causing an entry to be 
recognized as an individual handwritten character and treated accordingly. Compared to 
word recognition, individual character recognition is a relatively simple task for a typical 
state of the art handwriting recognition engine. Further, to increase the likelihood of 
presenting the user with the correct character, a pop-up list 132 is provided as shown in 
Figure 3B, listing likely characters in descending order of likelihood. This character list 
132 can be disposed in the same space as where the word list 130 would be when it is 
displayed, i.e. in the area 104 directly adjacent area 106. As each character is accepted, it 
is inserted at the end of an output string of characters (not shown) displayed above the 
input area 108. Optionally, a string of characters entered using the character writing area 
can be logged and automatically added to the user dictionary for future writing in the 
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word input area. If the character is being provided to correct a misspelled or 
misrecognized word, then the recognized character(s) are inserted at a selected position 
into the word, e.g., hap_y. 

Figure 4 is an example of a flow diagram of a method 140 for implementing the 
handwriting user interface of the preferred embodiment of the present invention. First, in 
step 142 a handwritten entry is made into the designated screen input area. In step 144 a 
check is made to determine whether the handwritten entry is in the text input area 104 or 
character input area 106, If the handwritten entry is in the text input area 104 in step 146, 
the handwriting recognition engine matches the handwritten input as described in Seni et 
al. Otherwise, in step 148 a probability score is generated for each possible character to 
indicate the likelihood that the input character is that particular character. In step 150, the 
highest scoring characters are identified. In step 152 the top scoring characters are 
displayed in a pop-up list 132. In step 154, the user selects the correct character and that 
selected character is inserted into or at the end of the character string in the input buffer 
and displayed above the input area 108. Then, returning to step 142, the user is allowed 
to continue character or word entry. 

Thus, the present invention provides a simple, easy to use interface for entering 
non-textual character strings in a handwriting enabled shirt-pocket sized device. 

While the invention has been described in terms of preferred embodiments, those 
skilled in the art will recognize that the invention can be practiced with modification 
within the spirit and scope of the appended claims. 



